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ABSTRACT 

One  application  of  distributed  computing  is  remote  system  instrumentation. 
Such  instrumentation  programs  require  good  response  with  low  overhead  to  pro¬ 
vide  timely  results  without  disturbing  the  system  being  measured.  A  remote  pro¬ 
cedure  call  system,  such  as  the  Circus  system  developed  at  Berkeley,  allows  pro¬ 
grammers  to  write  distributed  programs  with  little  more  effort  than  is  required  to 
write  local  programs.  This  paper  compares  a  Circus-based  implementation  of  a 
Berkeley  UNIXt  tool  (vmstat)  with  one  based  on  the  byte-stream  protocol  TCP. 
The  Circus  version  makes  for  much  cleaner  code,  but  it  requires  more  start-up 
time  and  higher  CPU  overhead  than  the  TCP  version.  We  conclude  that  the 
present  incarnation  of  Circus  is  not  acceptable  for  our  work,  but  that  future  ver¬ 
sions  of  Circus  should  prove  valuable. 


1.  Introduction 

One  application  of  distributed  computing  is  remote  instrumentation,  which  allows  a  user  on 
one  machine  to  monitor  the  performance  of  a  different  machine  without  logging  on  to  that 
machine.  Such  a  program  consists  of  at  least  two  processes;  a  data-gathering  server  process  on 
the  remote  machine,  and  a  data-displaying  client  process  on  the  local  machine.  If  only  one  server 
process  is  used,  it  multiplexes  connections  to  all  its  clients.  An  alternative  is  to  give  each  client 
process  its  own  server  process.*  In  either  case  the  communication  system  seen  by  the  client  and 
server  clearly  must  guarantee  the  accuracy  of  a  delivered  message.  In  addition,  we  feel  that  the 
communication  system  should  guarantee  message  delivery.  A  dropped  message  affects  the  client  as 
though  the  remote  machine  had  suddenly  slowed  to  a  crawl.  Thus  dropped  messages  would 
unnecessarily  annoy  users  and  possibly  confuse  analysis  programs.  This  problem  would  be  toler¬ 
able  if  messages  were  infrequently  lost.  Unfortunately,  casual  instrumentation  of  Ethernet  inter¬ 
faces  has  shown  input  error  rates  (hence,  dropped  packets)  of  50-100  or  more  per  hour,  which  is 
simply  too  high  to  ignore. 

Furthermore,  we  want  a  communication  system  that  is  general  enough  that  we  can  easily 
write  distributed  versions  of  existing  tools  or  write  new  distributed  tools  from  scratch.  However, 
the  communication  system  should  also  provide  adequate  performance  in  at  least  two  areas:  pro¬ 
gram  initialization  and  system  overhead.  Program  initialization  should  be  fast  because  we 
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sometimes  want  the  monitoring  took  to  tell  us  what  k  happening  nou;,  not  what  k  happening  10 
seconds  from  now.  Long-term  system  overhead  should  be  low  enough  that  the  took  can  provide 
relatively  long  traces  without  dkturbing  the  system  being  measured  (e  g.,  less  than  1%  of  the 
CPU  should  be  used  for  the  instrumentation  program,  and  it  should  not  cause  any  significant 
change  to  the  system’s  swapping  or  paging  behavior).  For  the  same  reason,  short-term 
overhead— which  we  will  not  consider  in  thk  paper— should  ako  be  low.  However,  short-term 
overhead  will  be  somewhat  higher  than  long-term  overhead  because  of  initialization  costs  and 
because  new  UNIX  processes  usually  get  higher  priorities  than  older  ones. 

One  reliable  and  general  communication  mechankm  k  the  remote  procedure  call  (RPC), 
which  by  and  large  allows  the  application  programmer  to  ignore  the  dktributed  aspects  of  the 
program.  Eric  Cooper’s  Circus  (Cooper  84a,  Cooper  84b|,  which  k  based  on  Xerox’s  Lupine  sys¬ 
tem  (Birrell  83],  k  a  remote  procedure  call  system  that  runs  under  Berkeley  UNIX  4.2BSD.  Circus 
differs  from  an  earlier  Berkeley  UNIX  RPC  system  (Larus  83]  in  that  it  k  based  on  datagram  ser¬ 
vice  rather  than  on  virtual  circuits. 

This  paper  evaluates  the  performance  of  a  Circus-based  version  of  vmstat^  by  comparing  it 
with  an  implementation  based  on  the  byte-stream  protocol  TCP  (TCP  8l|.  In  the  following  sec¬ 
tion  we  present  a  brief  introduction  to  what  a  programmer  works  with  when  using  Circus  and 
TCP  on  a  4.2BSD  system.  Section  three  describes  the  performance  tests  that  were  used.  The 
results  of  these  tests  are  described  in  section  four,  and  an  analysk  of  the  results  k  in  section  five. 
In  section  six  we  present  our  conclusions,  in  section  seven  we  suggest  additional  research,  and  in 
section  eight  we  summarize  the  paper.  Appendices  A  and  B  contain  sample  code  from  the  Circus- 
and  TCP-based  programs  (both  client  and  server),  with  an  emphask  on  the  differences  between 
the  two  approaches. 

2.  Programmer’s  View 

Circus  provides  the  UNIX  programmer  with  a  set  of  facilities  that  are  like  Lupine’s,  except 
that  some  changes  were  necessary  for  compatibility  with  the  Berkeley  UNIX  environment.  First 
the  programmer  defines  an  interface  of  types,  global  variables,  and  procedure  headings  using  a 
Mesa-like  language  derived  from  Xerox’s  Courier  specifications  (Mitchell  79]  (Courier  81).  From 
this  interface  the  rig  compiler  generates  C  code  (Kernighan  78]  for  the  server  and  client  stubs,  as 
well  as  a  header  file  that  contains  C  definitions  for  the  types  and  variables  specified  in  the  inter¬ 
face.  The  programmer  codes  two  programs,  one  for  the  client  and  one  for  the  server.  Taken 
together,  these  two  parts  differ  little  from  a  modular  non-dktributed  version  of  the  program. 
Most  of  the  differences  are  embodied  in  a  small  amount  of  code  that  manages  such  chores  as  bind¬ 
ing  the  client  to  the  server.  A  run-time  library  and  the  client  and  server  stubs  handle  communica¬ 
tion  between  the  client,  the  ringmaster  binding  process  (which  corresponds  to  Grapevine  in  the 
Xerox  world),  and  the  server.  A  programmer  using  Circus  ako  has  the  opportunity  to  program¬ 
matically  type-check  the  client/server  interface  with  the  UNIX  program  lint.  Relevant  portions  of 
the  Circus-based  vmstat  are  in  Appendix  A. 

As  part  of  its  Interprocess  Communication  (IPC)  facilities  (Leffler  83],  4.2BSD  provides  the 
UNIX  programmer  with  TCP  service.  In  contrast  with  using  Circus,  a  TCP-based  program 
requires  no  extra  paraphernalia  such  as  rig  (the  stub  compiler).  The  price  k  that  the  programmer 
must  do  more  work,  such  as  explicitly  opening  a  connection  between  the  client  and  server  and 
managing  I/O  errors.  To  handle  multiple  clients  simultaneously,  the  server  must  either  multiplex 
its  connections  or  fork  off  a  new  server  process  to  handle  each  new  client.  If  there  k  a  server  pn> 
cess  for  each  client,  then  the  client  bears  the  additional  burden  of  telling  its  server  to  exit  when  it 
(the  client)  k  ready  to  quit.  At  best,  thk  additional  work  k  merely  an  annoyance;  at  worst,  it 
provides  ample  opportunity  for  programming  mktakes.  An  additional  problem  with  using  TCP  k 
that  there  k  no  way  to  verify  the  type-correctness  of  the  client  and  server  communication  rou¬ 
tines,  other  than  checking  the  individual  read  and  u;r«fe  statements  by  hand  (which  k  ako  Uable 
to  mistakes).  Relevant  portions  of  the  TCP-based  vmstat  are  in  Appendix  B. 

*  viiBtit  produces  stitistics  sbout  the  virtusl  memory  subsystem  is  Berkeley  UNIX. 
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Thus  the  most  immediate  advantage  of  Circus  is  its  ease  of  use.  Another  expected  ^van- 
tage  results  from  Circus’  use  of  datagram  communication;  we  expect  lower  start-up  overhead  from 
using  Circus  than  we  do  from  using  a  byte-stream  protocol  Uke  TCP.  An  expected  disadvantage 
of  Circus  is  that  its  generality  may  make  communication  slower.  For  example,  Circus  aUows 
transparent  communication  between  different  machine  types,  which  may  le:^  to 
message-copying  or  format  conversion  in  the  case  where  both  machines  are  of  the  same  type.  A 
realistic  TCP-based  implementation  would  ako  have  to  deal  with  this  heterogeneity  problem,  but 
it  may  be  possible  to  hand-tune  the  communication  code  to  obtain  greater  efficiency  than  is  possi¬ 
ble  with  Circus. 

In  short  according  to  our  introductory  criteria  for  a  distributed  monitoring  tool,  we  expect 
that  Circus  would  make  an  excellent  tool  for  writing  remote  instrumentation  programs  if  we  could 
obtain  adequate  performance  from  it. 

3.  The  Teste 

We  propose  two  types  of  performance  tests:  one  test  measures  the  elapsed  start-up  time 
required  by  a  program;  the  other  test  measures  the  long-term  CPU  utilization  of  a  program.  T  e 
point  of  the  start-up  test  is  that  any  useful  instrumentation  utility  must  provide  quick  service 
without  high  initialization  costs.  The  point  of  the  utilization  test  is  that  any  useful  instrumentar 
tion  utility  must  not  significantly  disturb  the  system  it  is  measuring. 

Our  first  test  consisted  of  running  a  program  tha4  invoked  vmstat  300  timw  and  recorded 
the  accumulated  execution  time.  We  performed  this  test  on  a  VAX*  7M  with  2  megabytes  of 
physical  memory  running  in  single-user  mode.  In  one  case  we  performed  the  test  10  ^ 

Lmpeting  load,  and  in  a  second  case  we  performed  the  test  10  tunes  while  competing  with  a  load 

of  seven  simulated  “active"  users.^ 

The  second  test  consisted  of  causing  vmstat  to  iterate  (display  one  fine  of  statistics)  10.^ 
times  at  S-second  intervak.  When  the  test  finished,  both  the  client  and  the  server  "cord^^  infor- 
mation  such  as  their  elapsed  times  and  CPU  usage.  We  ran  this  test  7  times  on  a  VAX  780  with 
4  megabytes  of  memory,  at  various  hours  of  the  day  and  night  withoul^ention  to  machine 
load.  We  ako  ran  a  similar  test-using  20.000  iterations  instead  of  10, 000-to  ^ 

could  extrapolate  our  results  to  times  longer  than  a  day.  We  picked  ^ 

length  because  the  Berkeley  UNIX  kernel  updates  its  virtual  memory  statistics  at  1-  and  5-8«cond 
intervals  We  did  not  repeat  the  tests  using  a  1-second  interval  because,  as  we  shall  sw  in  the 
next  section,  the  CPU  utilization  at  the  5-second  refresh  rate  was  high  enough  that  additional 

tests  seemed  pointless. 


4.  Results 

The  results  of  the  first  test  are  summarized  in  Table  1.  Each  number  represents  an  average 
start-up  time  in  seconds.  We  also  repeated  the  start-up  tests  3  times  with  the  original  (single- 
process)  version  of  vmstat  for  rough  comparison  purposes. 


Table  1:  Start-up  times  for  TCP-  and  Circus-based  versions 


version 

with  load  no  load 

Circus 

TCP 

original 

4.63  1.36 

1.14  0.708 

10.2  2.04 

Table  2  gives  the  results  for  the  second  test.  We  obtained  these  numbers  by  compiling  code 
into  the  vmstat  client  and  server  so  that  each  program  recorded  its  elapsed  (“wall-clock  )  running 


*  VAX  is  1  trademark  of  Digital  Equipment  Corporation. 

4  Each  u«r  wa.  eimuUted  by  a  .hell  .cript  that  repeatedly  did  task,  such  as  compiUtKin,  editm*.  and  file 
copying. 
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.a  K. ...»  »a 

the  sum  of  the  CPU  time  used  iv  ..  J  \  Jq,  Circus  version,  and  we  ignore  certain 

r:r,ri;r.rr  -mS.’ -  -  •  "• 

original  vmstat  for  rough  comparison  purposes. 


Table  2:  Long-term  CPU  utilization 


system  time 
(sec) 

HSH 

%  of  system 
used 

Circus  client 
Circus  server 
Circus  (total) 

332.4 

437.4 

190.6 

123.6 

TCP  client 

TCP  server 
TCP  (total) 

50.3 

176.6 

108.7 

47.0 

0.32 

0.44 

0.76 

original  (total) 

97 

117 

0.44 

5.  AnalyaU 

Having  seen  these  results,  we  now  must  interpret  them. 

6.1.  Start-up  Delay  program  consistently  starts  up  faster  than  the 

Table  1  shows  that  the  T  as  bvte-strcam  protocols  have  a  reputation  for 

Circus  version.  This  result  may  ‘^HowJUr  the  protocol-related  activities  may  only  be  a 

high  overhead  in  establishing  connections.  However  P  ^ 

small  part  of  all  the  program’s  actmti^  30  ^s  (Vc  of  its  total  CPU  time)  estate 

one  run  of  100  iterations,  the  TCP-bas^  't  "  uiJ^  less  than  10  ms  (0.2%)  of  CPU  time  to  con- 
Ushing  a  connection;  the  Circus-based  client  J„d  a  message  to  rin,- 

nect  to  the  server.  However,  to  6nd  » g,rof  and  is  presumably  slow. 

matter  and  then  wait  for  a  repy,  w  i  u:  i,  goends  100  ms  looking  up  the  server’s  Internet 

Contrast  this  with  the  TCP-b«ed  version  w^cb  sp.n^  M  ^ 

cp^ ... ...... .... - ...... 

... 

because  the  original  version  experimental  versions  do  only  one  nlist,  when  the 

memory,  .s.h  t.m.  .1  »  Both  ot  th.  .xp.riin..  »  Thus  th.  .om- 

r -rhf.::"  ^h^;  .t."  u.'r.rp."rr.. .  h.....  h.. ..  p.....  ... .. ......... .. 

Ssins  .h.  C.../......  P.r.'iis"'  .....um.n....ou 

R  i  CPU  utilisation  and  Steady-State  Delay 

OaZa  KyrKJ  V^w***»*«»  ak*  TPP  v^r^ioD*  it  uses  about 

As  with  the  6rst  test,  the  are  many  causes  for  this  difference, 

3  times  as  much  of  the  CPU  «  of  ^hich  result  from  Circus’s  design,  and 

some  of  which  result  from  ’t^m  is  that  it  must  send  a  message  to  the 

level.  One  inherent  problem  of  the  WC  ^  TCP-based  system,  the 

server  for  every  information  message  ‘bat  th  message  ending  the  link.  Gprof 

server  just  keeps  sending  up  to  14%  of  its  time  (ie.,  0.14%  of  the 

analysis  suggests  that  the  Circus-based  client  can  spend  «P byte- 

^o^ylUtstMcl^:^^^^^^^^^^  TcTblsed  version)  in  parameter  passing.  This  copying  comes 
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from  moving  whole  structures  around;  the  TCP-based  version  just  moves  pointers. 

There  is  another  problem  that  is  not  inherent  to  RPC  systems  in  general  but  which  results 
from  Circus’s  charter  as  a  reliable  remote  procedure  call  system.  Circus  allows  more  than  one 
process  to  export  a  given  service.  When  a  client  makes  a  remote  caU,  the  cUent  stub  sends 
requests  to  all  servers  that  export  that  call  and  uses  a  voting  scheme  to  determine  the  result  that 
it  returns.  Circus  does  provide  a  mechanism  that  allows  the  client  to  specify  which  server  (or 
servers)  to  use.  However,  the  techniques  necessary  to  handle  communication  in  the  general  case 
(multiple  servers)  can  have  appreciable  cost  even  when  only  one  server  is  called.  Thus  remote 
instrumentation  programs,  which  do  not  need  this  repUcation  mechanism,  must  bear  this  added 
cost  when  using  Circus. 

The  lack  of  tuning  in  Circus  leads  to  problems  such  as  unnecessary  malloc  (memory  alloca¬ 
tion)  calls,  expensive  queueing  operations,  and  unnecessary  copying.  The  mallocs  are  done  at  each 
call  when  the  client  stub  allocates  and  returns  buffer  space.  The  stub  could  avoid  these  problems 
by  maintaining  its  own  pool  of  buffers.  The  queueing  operations,  which  support  communication, 
could  probably  be  made  less  expensive  by  using  register  variables  (Kernighan  78).  Although  any 
general-purpose  communication  mechanism  must  provide  machine  independence,  it  seems  reason¬ 
able  for  the  stubs  to  recognize  that  they  are  running  on  compatible  architectures  and  agree  to  use 
that  architecture’s  data  format,  rather  than  wasting  cycles  converting  to  and  from  some  general- 
purpose  format.® 

There  are  two  good  reasons  for  putting  Circus’s  reliable,  procedure-oriented  communication 
protocol  in  the  kernel.®  Because  Circus  runs  entirely  in  user  space,  it  must  implement  timeouts 
using  the  alarm  library  routine,  which  means  that  Circus  preempts  SIGALRM  signals.  The  first 
problem  is  that  this  preemption  forces  users  who  want  an  alarm-clock  function  to  use  an 
inefficient  kludge.  The  second  problem  is  one  of  performance.  When  the  stub  sends  off  a  request, 
it  must  make  at  least  four  system  calls:  the  first  call  sends  the  request,  the  second  call  sets  the 
alarm,  the  third  call  (select)  waits  for  a  reply  from  any  of  the  servers,  and  the  fourth  call  finally 
reads  in  the  result.  Each  additional  server  requires  two  additional  system  calls.  A  kernel-based 
implementation  of  Circus  would  avoid  both  of  these  problems. 

0.  Concluslona 

The  purpose  of  the  preceding  tests  was  to  evaluate  a  Circus-based  remote  instrumentation 
program.  These  tests  lead  us  to  conclude  that  Circus’s  current  incarnation  is  not  ready  for  use  in 
production  programs.  The  problem  is  not  the  slower  elapsed  start-up  time  of  the  Circus-based 
version,  which  is  negligible  (and  certainly  faster  than  the  original  version  of  vmstat).  A  more  seri¬ 
ous  problem  is  the  CPU  overhead  that  Circus  requires,  which  is  twice  our  1%  guideline.  For¬ 
tunately,  tuning  the  performance  of  Circus,  removing  the  replication  mechanisms,  and  moving  the 
communication  code  into  the  kernel  should  solve  this  problem.  We  predict  that  once  this  task  has 
been  done.  Circus  will  be  an  excellent  tool  for  distributed  monitoring  programs.  In  the  meantime, 
programmers  will  have  to  balance  their  desperation  for  such  a  distributed  program  against  the 
pain  of  writing  a  program  based  on  TCP. 

7.  Additional  Research 

While  this  paper  provides  generally  encouraging  results,  additional  work  should  be  done  to 
confirm  our  optimism.  In  particular,  we  would  like  to  repeat  these  tests  using  the  non-rcplicated, 
kernel-based  version  of  Circus  being  developed  by  Karen  White  [White  85],  after  it  has  been  as 
thoroughly  tuned  as  the  Berkeley  UNIX  TCP  implementation. 

Furthermore,  there  are  metrics  other  than  the  ones  that  we  have  chosen.  The  most  obvious 
candidate  for  additional  testing  is  memory  usage.  A  server  with  a  large  working  set  size  (e.g., 

B  The  dericn  for  a  new  vereion  of  CSrou  ineludee  thb  ftub-to-ftub  hindshnke  »nd  »  Ex  for  the  buffer  epece 
problem. 

•  “relUble”  u  in  "fuiruiteed  delivery  of  uncomipted  d»t»,”  not  u  in  “replicated.” 
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from  buffer  requirements)  will  certainly  disturb  the  system  which  it  is  trying  to  measure,  even  if 
its  measured  CPU  utilization  is  low. 

8.  Summary 

Having  identified  an  interesting  class  of  distributed  computing  programs  (remote  instrumen¬ 
tation),  we  have  decided  on  certain  performance  requirements  and  a  possible  technique  for  writing 
programs  that  belong  to  that  class  (remote  procedure  calls).  We  have  built  a  realistic  example 
program  using  this  technique  (by  modifying  vmstat),  and  we  have  obtained  encouraging  results  by 
comparing  this  example  program  with  a  version  based  on  a  different  technique  (TCP).  We  expect 
that  as  Circus,  a  UNIX  implementation  of  this  technique,  is  refined,  it  will  compare  favorably  with 
its  competitors. 
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Appendix  A:  portions  of  the  Clrens-bnsed  rmstat 

-  G(i^)vm»tat.ri$  l.t  10/18/84 

vmsUt  "  be^n 

-  The  following  Hock  it  from  <iyt/vTnmeler.k>: 

vmmeter  type  “  record  [ 

v_rwrtch:  loBg  CArdiaal, 
v_trtp:  long  CArdiaal, 
v_syscEU:  ioag  cardiaal, 
v_mtr:  loag  cardiaal, 
v_soft;  loBg  cardiaal, 
v_pdma  loag  cardiaal, 
v_pswpm:  loag  cardiaal, 
v_pswpout:  loag  cardiaal, 
v_pgui;  loag  cardiaal, 
v_pgout:  loag  cardiaal, 
v_pgpsm:  loag  cardiaal, 
v_pgpgout:  loag  cardiaal, 
v_mtruir.  btag  cardiaal, 
v_pgrec:  loag  cardiaal, 
v_xsfrec;  loag  cardiaal, 
v_xifrec:  loag  cardiaal, 
v_exfod:  loag  cardiaal, 
v_rfod:  loag  cardiaal, 
v_vrfod:  loag  cardiaal, 
v_Dexfod:  loag  cardiaal, 
v_Dzfod:  loag  cardiaal, 
v_nvTfod:  loag  cardiaal, 
v_pgfrec:  loag  cardiaal, 
v_riults:  loag  cardiaal, 
v_scaii:  loag  crardiaal, 
v_rev;  loag  cardiaal, 
v_8«qfree:  loag  cardiaal, 
v_dfr«e:  loag  cardiaal, 
v__ri£tpgTec:  loag  cardiaal, 
v_swpin:  loag  cardiaal, 
v_swpout:  loag  cardinal 


vmtotkl:  type  —  record  [ 
t_rq:  iateger, 
t_dw.  iateger, 
t_pw:  integer, 
t_sl:  integer, 
t_sw:  integer, 
t_vin;  long  iateger, 
t_avin;  long  iateger, 
t_rm;  integer, 
l_trm;  iateger, 
t_vmtxt:  long  integer, 
t_avTntxt:  long  integer, 
t_rmtxt:  iateger, 
t_irmtjrt:  integer, 
t_rre«:  iateger 


-  (mi  of  <iyi/vmmeteT.k>) 

-  The  following  block  it  from  <tgt/vmtgttm.k>: 

forkstet:  type  —  record  ( 

entfork;  loag  integer, 
cntvfork:  long  integer. 


1; 


sizfork:  loBS  iakescr, 
sizvfork:  loag  imteger 


(md  of  <tyt/vm$yttm.k>) 


doubleFlo»t;  type  —  array  2  of  l< 

CPUSTATES:  iatcger  —  t; 
DK_NDRrVE;  iatcgcr  —  t; 

VMSTATS;  type  —  record  ( 
busy:  loaf  iateger, 

time:  array  CPUSTATES 
xfer  array  DK_NDRIVE 
R»te:  vmineter, 

Total:  vmtotil, 

Sum:  vmmeter, 

Forkstat:  forkstat, 
rectime:  loag  eardiaal, 
pgintime:  loag  eardiaal 

1: 

disk_drive:  type  —  record  ( 
name:  atriag, 
unit:  iateger 

1: 

Time_t:  type  —  loag  iateger; 

vm  init:  type  *  record  [ 

drive:  array  DK_NDRrVE 

bx:  loag  iateger, 
phx:  loag  iateger 

1: 

vin_iiifo:  type  —  record  [ 
time:  Time_t, 
s:  VMSTATS, 
deficit:  loag  iateger, 
etime:  doubleFloat, 
nintv:  loag  iateger 

1: 


iBg  iateger; 

-  from  <$yt/dk.k> 

-  from  <$yt/dk.k> 

_  package  of  virtual  memorp  otati 

of  loag  iateger, 
loag  iateger. 


-  info  for  iitk  drive* 


-  oeeondt  oinct  IfanTO 

-  initial  meotage  to  dimt 

of  disk_drTve, 

—  dock  rate 

—  profiling  dock  ratef 


—  regular  VM  otato  info 
-  time  tkat  number*  were  gotten 

-  antidpoted  memerg  deficit 

-  otat*  eolleeiion  interval 


vTiBt»t_info:  proeedare  [TirstCelL  booleaa]  retaraa 

pinfoPkt:  vin_mfo]; 

vni£t»t_mit:  proeedare  retaraa  [initPkt:  vm_mit); 
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#ifa<icf  lint 

static 

#eadif 

char  ^ccsid2  —  "0(#)vniBtst.e 

#iBclB<ic 

<sys^u»in.h> 

f^iaclade 

<stdio.h> 

#iaelade 

<strmgs.h> 

^iaclade 

<sys^Tn.b> 

^iaclade 

<sys/dk.h> 

#uiclade 

"  vTnst»t_defs.h* 

#defiae 

HOSTNSIZE  255 

f^efiae 

YES  1 

#deriae 

NO  0 

iat 

HZ; 

char 

bost[HOSTNSlZE]; 

aasifaed 

stime; 

vin_init 

initPkt; 

vin_info 

infoPkt; 

^Mefiae 

INTS(r)  ((x)  -  (bi  +  phi)) 

2.7.1^  (Irupfery^est)  4/lt^’; 


/*  ko$t  name  aize  */ 


/*  name  of  the  hoot  we  want  to  tath  to 
/•  Sleep  time  between  refreokea  */ 

/*  init  packet  */ 

/*  info  packet  for  one  call  */ 


V 


•  Print  VM  atatietica,  nainf  a  remote  aerver  to  eoUeet  the  daU. 

*  Vaea  Eric  Cooper" a  Circua  for  RPC. 

V 

main(argc,  irgv) 

int  iTgc; 
eh*r  *Vjv; 

int  lines;  /*  keaderinf  */ 

lot  iter;  /*  nurriber  of  itcrationa  to  make  */ 

eztera  char  _sobuf[]; 

boolein  nrstCall;  /*  flag  S»f  Iri  / 

If  (u-gc  >  1  tic  strcmp(ugv[l],  t”)  0)  { 

set_lr»ce_n»gs(  »rgv[2)); 
irgc  — =■  2; 

»rgv  +»-  2; 

} 

setbuf(stdout,  _sobuf); 

stime  -»  5;  /*  dcfavlt  aieep  time  */ 

(void)  gethostaiiTie{host,  HOSTNSIZE); /*  default  hoat:  «»  */ 

»rgc - ,  iTgv++; 


*  Fijvre  out  how  many  itcrationa  to  make  and  how  long  for  each 

*  refreak.  If  no  numbera  were  given,  orUy  iterate  1  time.  If 

*  only  a  refreak  interval  waa  given,  iterate  forever.  Otherwiae, 

*  the  uaer  will  tell  ui  kow  many  timea  to  iterate. 

V 

if  (wgc  <  1) 

iter  ™  1; 

clac  { 

stime  »toi(»rgv|0]): 
if  (»rgc  1) 

iter  -•  0;  /*  eloae  enough  to  infinity...  */ 

cIm 

iter  ■■  »toi(argv(l]); 


main 
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•  Import  the  vmttat  inlerfuee  and  gel  1—titne  info  abent  tie  dock 

*  rate  and  drivet. 

V 

»et_troup«_list(l,  host); 

If  (!iniport_vinst»t())  { 

fprintf(stderT,  ’e»n't  import  vnwt»l\i»’’); 
ejit<l); 

} 

initPkt  “  vnist»t_mit(); 

HZ  —  initPkt.phi  T  initPkt.phi  ;  initPkt.hi; 
nrstCill  —  YES; 


reprint: 

lines  “  20; 

printf('\ 

proct  mentory  9&— 18.18s  pa^e  disk  faults  cpu\o\ 

r  b  w  avm  fre  re  at  pi  po  fr  de  sr  %c%d  ^Bc%d  %c%d 

host, 

initPkt.drivelOl.name(0],  initPkt.drive|Ol.unit, 
initPkt.driveilj.naineloi,  initPkt.driveilj.unit, 
initPkl.drivel2j.name[0l,  initPkt.drive(2i.anit, 
initPkt.drive(3i.nameio],  initPkt.drive[3l.unit); 


in  sy  cs  us  »y  id\n". 


kwp: 

infoPkt  ™  vn«tat_info(nr8tCan); 
nrstCall  —  NO; 

displayinfo(&infoPkt); 
if  ( — iter  —  0)  { 
ejdt(0); 

go_to_sleep{stin»,  0);  /*  (Zero  tnieroieeondt)  */ 

If  ( - lines  0) 

(oto  reprint; 

(oto  loop; 

) 


-  12  - 


#irBder  lint 
•iatic  cliAT  sccsidQ 

#cBdir 


'’0(#)vTiEtatd.e 


S.8.1.2  (lnipfer/«t)  4/14^’; 


<stdio.h> 

<strings.h> 

<sys/^octLh> 

<sys/tir»m.h> 

<sys/ile.h> 

<sy?/irin.h> 

<sys/dk.h> 

<nlist.h> 

<sys/^uf.h> 

<sys/^»it.h> 

<sysyd.ime.h> 

<sys  /^esource.h> 


l^iBcladc 
#{Bcladc 
#iBcladc 
^iaclade 
#iDclade 
#iaclnde 
#iacl«dc 
#iaelBde 
#uiclnde 
^uicladc 
#UicIadc 
#iaclade 
#ifdef  viLX 

#iaclade  <vuruba/ibavu'.h> 
^iaclnde  <vijanbt/^btvar.h> 

#«adir 

^inclnde  *vTiistit_defs.h" 

#defiae  YES  1 

fdefiaa  NO  0 


/• 


*  program  timely  provides  rubroutineo  wkick  tke  Hiwi  esUo  vis  «n 

*  RPC  rTvechanitm.  Vmstai_init  if  called  once  (per  client),  •«  (Act  we 

*  don’t  waste  time  retransmitlint  repetitious  information  (ey.,  dock 

*  rate).  After  tkat,  vmstai_info  is  called  every  lime  a  dient  wants  new 

*  information. 

V 


«ai«ra  iat  ermo; 
aacisaed  stirmn:!; 


time_t 

vTn_info 

vm_info 

vin_init 

vm_mfo 

tinK_t 

time  t 


boottime; 

cvginfo; 

runninglnfo; 

initPkt; 

inToPkt; 

listRefresh^); 

now, 


/*  interval  between  refreskes  */ 

/*  6oo((im«  to  now  averages  */ 
/*  running  rales  */ 

/*  packd  of  init  info  */ 

/*  packd  of  regular  info  */ 

/*  time  of  last  refresk  */ 
f*  (tke  current  time)  * / 


m^(irgc,  trgv) 

iat  nrgc; 


main 


char 

{ 


rtvO: 


iat  k 

time_t  initiilizeO; 

if  (irgc  >  1) 

stime  —  »toi(crgv[l]); 
boottime  miti»lize(); 
refreshO; 


f*  gd  dock,  disk  drive  info  */ 
/*  read  1st  sd  of  values  * / 


if  (!export_vni6tct())  { 

fprintf(stderr,  "can't  export  vnist»t\n"); 
exit(l); 

> 

/* 

*  Disassociate  oursdves  from  our  parerd.  Tkis  is  espeetally 

*  needed  if  you  use  rsk  to  start  up  tke  server, 

V 

^Kfadcf  noOrphan 


IS  • 


^■dif 

) 


r  •••  V 

server  k»p(); 


/*  /  tk,  krmd  tni  io  one— time  readini  of  kemd 

•  Get  the  nameiitt  for  the  kemd  ont  t 

•  memory.  Return  tke  eyetem  hoot  t.me,  enrf  eet 

•  time. 

V 

tiine_t 

initialize() 

{ 

r  •••  V 

} 

Send  iniliol  dot.  to  tke  dienl:  dock  rote  ^  .  Ion, 

*  Be  eure  to  updote  our  buffer,  ./  ««  koven  t  hem  tolled  m  ton, 

*  time. 

V 

vin_mit 

^  if  (timK&now)  -  l»stRefresh  >—  «»"*)  { 

lastRefresh  *  now; 
rcfreshO; 

} 

•  pt-e  avers, «  n«m6ers  (overoyed  sinee  system  ftoot;.  OtAenwse, 

*  tke  server  ikould  give  tke  join/  rate. 

V. 

VTn_mfo 

o(rirstciU) 
boolean  nrstcall; 

^  if  (tinie(&now)  -  lastRefresh  >—  stime)  { 

lastRefresh  “  now, 
refreshO; 

> 


If  (nrstcall) 


else 


bcopM&avglnfo.  tinfoPkl.  #i»eoHvni_inro)): 
bcopyl&runnmgliiro,  fcinToPkt,  .laeoHvTn_mro)); 


ret«r*(infoPkt); 

) 

Re/res*  tke  '.v,/n/a'  ortd  ’runninglnfo^  buffer,.  U,e.  tke  ylobol 
*  etirrenl  time  Cnow’). 

V 

refresh() 

^  tin*_t  interval; 

interval  “  now  —  boottune;  .  ,  , 

getinfo(mterval,  feavglnfo,  ftninninglnfo); 

} 


initialize 


vmstat_init 


vmstatjinfo 


refresh 
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Appendix  Bt  portions  of  the  TCP-bmsed  Tmetat 


/*  QfH^Jvmttat.k 


s.e 


S/It/Si  •/ 


#denac  VMSTAT_EXrr  'X' 

#denec  ERRLOG  "vmstatd.errlof" 


/*  me»$age;  kiti  the  atrrent  emneetion  */ 
/*  error  tog  for  the  eerver  daemon  * / 


typedef 

{ 


•track 

iat  busy; 

loBC  timejCPUSTATES]; 

lo>s  xfer[DK_NDRIVEl: 

•tract  vmmeter  Rite; 

•tract  vmtotkl  Totsl; 

•tract  vmmeter  Sum; 

•tract  foricstmt  Forkst4l; 

aa^igacd  rectime; 
aB^icaefl  pgintime; 


}  VMSTATS; 


art  alt  lent  to  the  elient  at 


*  The  variaUet  in  the  following  block 

*  one  time  or  another. 


ctar  dr_n»me[DK_NDRIVE](lO]; 
ckar  dr_unit(DK_NDRIVE|; 


iBt 

phz; 

iat 

hz; 

VMSTATS 

r, 

lime_t 

now, 

iat 

deficit; 

doable 

etime; 

aaaigaed 

stime; 

iat 

nintv; 

#denBc 

rate 

s.Rate 

#dcfiac 

total 

s.Totol 

#denac 

sum 

s.Sum 

#dcnac 

forkstat 

s.Forkstat 

/•  time  that  we  read  from  /dev/kmem  */ 

time  at  tpedfied  bg  command  line  */ 
—  bootlime  [let  part  only)*/ 


/*  tleep 
f*  now 


*  INTTBVFSIZE  it  the  ntim  of  bylet  needed  to  buffer  tke  initi^ization  data: 

*  dr_name,  dr_unit,  phz,  and  ht. 

*  MESGBUFSIZE  it  the  num  of  bylet  needed  to  buffer  one  mettage: 

*  now,  t,  deficit,  dime,  and  nintv. 

*/ 

#denBe  INITBCFSIZE  (10*DK_NDRrVT*^lieoHch»r)  +  DK_NDRrV'E*»lieof(char)  \ 
+  ■iseofiat)  +  aiseofliBt)) 

^deHne  MESGBUFSIZE  (•i*eor[tiine_t)  +  •iieof^VMSTATS)  +  aiieoftiBk)  \ 

-t-  •iieof\donble)  +  •iieof^iat)) 

char  vms_’mitbuf[INlTBUFSlZE]; 

char  vms_mesgbuf[MESGBUFSIZE); 


IS  - 


#irBder  lint 

static  ekar  Vesid2  » 

#eBdir 

^iaclade  <sys^»r»in.h> 
#iaclBde  <signiLb> 
f^iacladc  <netdb.h> 
#iacladc  <stdio.h> 
#iBclads  <strings.h> 
#iaclBds  <3ys/^rm.h> 
#iBclBde  <sys/dk.h> 
^iBclBdc  <sys/fe>cket.h> 
^iaclBdc  "vmst&t.b’' 
#deria«  HOSTNSIZE  100 
#deriae  YES  1 
#deriac  NO  0 


*0(#)vmrt»t.e 


2.8  (loipferyd«t)  K/ii/UT', 


/•  kttl  nsrrM  nzt  */ 


lat  HZ; 

ckar  bost[HOSTNSIZE); 

iat  sock; 

#denac  lNTS(x)  ((x)  -  (bi  +  pbt)) 


/*  name  of  tke  koot  we  want  to  talk  to  */ 
/•  toekA  for  talkini  to  tke  server  */ 


r 

*  Ptint  VM  otatiotiea,  utinf  a  remote  server  to  eolleet  tke  data. 

V 

msin(irge,  srgv) 

iat  srgc; 
ckar  *^gv; 


{ 


/•  count  tinea  for  keaierini  */ 

/•  number  of  iterationo  to  make  */ 


iat  lines; 
iat  iter; 

extcra  ckar  _sobufIl; 
street  bostent  ^p  *  NULL;  /*  pwnis  to  koA  deaertfkion  */ 

iat  sockopen();  /*  opena  a  aoekA  connection  to  tke  aerver  •/ 

iat  quit(); 


(void)  signil(SlGINT,  quit); 

(void)  sign»J(SlGHUP,  quit); 
setbuf(stdout,  _sobuf); 

stime  «  S; 

(void)  getbostn«iie(bost,  HOSTNSIZE); 
host[HOSTNSIZE-l]  —  XO'; 
bp  “  getbostbyniiTie(bost); 
srge - ,  argv+-b; 


/•  otkerwiae  aerver  dumps  cere  •/ 


/*  (default  »  tost  toe 're  on)  •/ 


/' 


*  Figure  out  how  many  iterationa  to  make  and  how  long  for  eaek 

*  refreah.  If  no  numbera  were  given,  only  iterate  1  time.  If 

*  only  a  refreak  interval  waa  given,  iterate  forever.  Otkerwiae, 

*  the  user  will  tdl  us  how  many  limea  to  iterate. 

V 


if  large  <  1) 

iter  “  1; 

«ls«  ( 

stime  ™  »toi(»rgv|0]); 
if  (arge  —  1) 

iter  ■■  0; 

else 

iter  atoi(argv(l]); 

> 


/*  doae  enough  to  infinity...  * / 


sock  “  80ckopen(bp,  "ymstat"); 


main 


-  IS  - 


*  Tell  tke  $erver  how  lonf  to  tleep  ond  ftt  initial  ( 1-time) 

*  info. 

V 

writ«(sock,  (char  •)&stiinE,  •iaeof  rtime); 
recv_init(sock): 

HZ  —  phs  T  phi  :  hi; 


reprint: 

priiitf("\ 
procs 
r  b  w 


lines  •  20; 

memory  %-18.18s  pi«e  disk  faults  q>u\n\ 

avm  fre  re  at  pi  po  fr  de  sr  %c95d  %c%d  %c%d  %c%i 


host, 

dr_name|0][0],  dr_unit[0],  dr_name[ll[Ol,  dr_unit[ll, 
dr_name[2j[0],  dr_unil[2i,  dr_name[3l[0],  dr_unit|3]); 


in  sy 


a  iia  gy  id\n", 


kx>p: 


) 

r 

*  Coteh 

V 

quit() 

{ 


} 


recvinfo(sock); 

displayinfa(); 

if  ( - iter  —  0)  { 

cleanup{sock); 

exit(0); 

} 

if  ( — lines  <—  0) 

goto  reprint; 
goio  loop; 


s  S/G/IVT  end  r^it. 


cleanup(sock); 

exit(0); 


quit 


*  Tdl  tke  eerver  that  we  ere  done;  flueh  snp  dots  remaininy  in 

*  the  connection. 

*/ 

char  quitnowfl  —  VMSTAT_EXIT;  /*  mettett  to  dote  the  connection  */ 

cleanup(asock)  .  ,  / 

iat  asock;  /*  ***  socket  we've  been  nnng  / 

^  char  fbufltOfiS];  /*  buffer  for  fluthing  the  eennedien  */ 

if  {send(asock,  quitnow,  #i«eof[quitnow),  MSG_OOB)  <  0)  { 
perror("vmstat:  sending  quitnow"); 
exit(l); 

while  (read(isock,  fbuf,  eiacof  fbuf)  >  0) 

( 

(void)  close(  asock); 

) 


cleanup 
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#irBder  lint 

•t»tic  ekw  sccsidfl  —  "0(#)vnst»td.c 

#eBdif 


<si{naLh> 

<sys/tiiiie.h> 

<sys/ile.h> 

<Detdb.b> 

<stdio.h> 

<sys/octLh> 

<sys/^ir»in.h  > 

<netmet/n.h> 

<errno.h> 

<sys/4ocket.h> 

<sys^rm.h> 

<sys/dk.h> 

<nlist.h> 

<sysy^uf.h> 

<sys/w»it.b> 

<  sys  y^esource.b  > 


^iacladc 
^{HatJadc 
#iBclade 
#{Bclada 
#iBelade 
#iaclnde 
#iBclade 
#iaclnde 
^iBclnde 
f|liiicladc 
#iBclade 
#iBcIad« 
iPiBclade 
#iacladc 
#iBcladc 
#iacladc 
#irder  vu 

#iBcIadc  <vuubt/^ibaw.b> 
#iBclada  <vuanbk/4Tibavu'.b> 

#«Bdir 

^iaclada  <strings.b> 
#iBclad«  ’vmstat.b* 

#denBC  YES  1 
#dcfiB«  NO  0 


8.0  (lnipfer/«t) 


*  Thi$  protram  ereatei  a  aaeket,  doet  $ome  reaiint  of  ike 

*  kernel  memory,  and  tken  befini  an  infinite  loop. 

*  Eaek  time  throufk  the  loop  it  aeeepte  a  connection  and  forko  off 

*  a  child  to  manate  it.  The  chUd  oimply  lende  vmotat  numbere  thru  it. 

*  Wbm  the  dient  tdle  the  child  to  fuit,  the  child  doeeo  the 

*  old  connection  and  erite. 

V 

iat  msgsock; 

cxiera  iat  eimo; 

nuunO 

{ 

iat  i; 
iat  sock; 

iat  re»per()l 

time_t  boottime; 
tiine_t  initialize(); 
iat  servsock(); 

sock  “  servsock(”vinst»t"); 
boottime  —  initiiliie(); 

(void)  sign»l(SIGCHLD,  reaper); 

r 

*  Dieaeeoeiate  ouredveo  from  our  parent.  Thio  io  topeeeally 

*  needed  if  you  uee  reh  to  atari  up  the  server. 

V 

iPifadcf  DoOrpban 

/» ...  V 

#cadif 


*  Start  aeceptinf  conneetiona.  The  aceejt  miiht  fail  if  we  fd 

•  interrupted  by  a  chid" a  exitinf.  If  thio  happena,  juat  try 

»  again.  In  ease  of  an  unexpected  error,  we  pauae  firat  before 


/*  the  aoekei  eonneetinf  with  the  dimt  */ 

/*  reaps  exit'd  ehUd  processes  */ 

/•  sets  up  a  aoekei  for  the  server  */ 


main 
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*  rHryinj.  There  i$  €  B—foli  metive  for  tkio:  (l)  given  oome 

*  breathing  room,  magbe  the  kernel  will  gel  ito  act  together  and 

*  then  we  won't  gel  the  oame  error  again.  (S)  at  leait  the  error 

*  log  file  won't  grow  to  rapidlg. 

V 

(void)  listen  (sock,  6); 

for  (;;)  { 

msgsock  »ccept(sock,  (nirBet  socktddr  (iat  ^); 
if  (nsgsock  <  0)  { 

if  (errno  1—  EINTR)  { 

perTor(*vinBtatd:  iccept"); 
sleep  ((aasicacsl)  S); 

} 

eeatiaac; 

} 

if  (fork()  —  0)  { 

(void)  ck»e(sock);  /*  (child)  */ 

doTheDirt}rWork(boottiine); 

) 

•Im 

(void)  ck)8e(m8(90ck); 

} 

} 

r 

*  Watt  until  a  ehUd  preeett  emit. 

V 

re»per() 

{ 

aaioa  wait  status; 

while  (wait3(i:status,  WNOHANG,  (etraei  rusage  *)0)  >  0) 

> 

r 

*  Gel  the  nameliit  for  the  kernel  and  do  ang  one-time  reading  of 

*  kernel  memory.  Return  the  lyttem  boot  time. 

V 

tinie_t 

initialize() 

{ 

/*  ...  V 

> 

/• 

*  The  ehUd  firtt  readt  the  ileep  time  (ie.,  the  tampling  rate)  and 

*  sends  initial  information  tuck  at  Ike  device  names,  (ke»>  *unit*i, 

*  and  tome  dock  info.  It  then  reealeulatet  how  long  it’i  been  linee 

*  tyttem  boot  (thii  it  used  once,  to  that  we  can  get  the  long-term 

*  average  ratet  tinee  boot  time;  after  that,  we  use  the  current  rates/ 

V 

doTheDirty  Work(  boottime) 

tiine_t  boottime;  /*  tyttem  boot  time  */ 

^  iat  oob();  /*  eatcAes  SIGURG  •/ 

(void)  signal(SIGURG,  oob); 

(iat  pid  —  — getpid(); 

(void)  k>ctl(nBgsock,  SIOCSPGRP,  (char  *)&pid);) 
send_init(); 

read( msgsock,  (char  ^&5tinie,  aisaof  stime); 


reaper 


initialize 


doTheDirtyWork 


(void)  time(&now)., 
nintv  B  DOW  —  boottime; 


IS  - 


for  (:;)  { 

CetinfoO: 

•endinfoO; 

$leep(stime); 

nintv  —  I;  /*  (f»r  fHinfo’t  fkt)  */ 

} 

) 


•  CatcA  ■  SIGURG:  rt»i  in  tni  proeeat  «  maiat*  •**  dient. 

V 

oob() 

ch»r  &buf[l]; 


(void)  recv(msgsock,  abuf,  •iacof  abuf,  MSG_OOB); 

■witck  (&buf[0|)  { 
cmM  VMSTAT_EXrr: 

(void)  ckMe(mpoek); 

exit(0); 

brook; 

defoolt: 

fprintf(stdeiT,  "vmstatd:  unknown  requ«t;  0%o\n*,  kbufJOJ); 

brook; 


) 

) 


•  Paekate  the  eoUeeted  info  in  a  buffer  and  tend  H  off  to  the  dient. 

V 

aendinfoO 

ebor  ^ufp  “  vn»_n*sgbuf;  /*  pointo  into  the  buffer  */ 


bcopy((ehor  *)fenow,  bufp,  aiteof  now); 
bufp  +>“  oiaeof  now; 
bcopy((chor  *)&s,  bufp,  siacof  s); 
bufp  siacof  s; 

bcopy((chor  *)&dericit,  bufp,  siacof  deficit); 

bufp  +“  siacof  deficit; 

bcopy((ehor  *)&etime,  bufp,  siacof  etime); 

bufp  siacof  etime; 

bcopy((chor  •)&nintv,  bufp,  siacof  nintv); 

write(msgsock,  vii»_mesgbuf,  MESGBUFSIZE); 

> 

/* 

*  Move  the  device  nameo  and  dock  info  into  a  buffer  and  then  tend  it 

*  all  off  to  the  dient. 

V 

send_mit() 

^  *bufp  —  vn»_mitbuf;  /*  pointo  into  the  buffer  •/ 


beopy(dr_unit,  bufp,  siacof  dr_unit); 
bufp  +w*  siacof  dr_unit; 

bcopy(&dr_n»inel0]10],  bufp,  siacof  dr_n»ine); 
bufp  +—  siacof  dr_n>nie; 
bcopy((chor  ^kba,  bufp,  siacof  hi); 
bufp  siacof  hi; 
bcopy((chor  *)&phi,  bufp,  siacof  phz); 

write(nvg90ck,  vmB__^initbuf,  INTTBUFSIZE); 


oob 


sendtnfo 


send_init 


} 


